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PREFACE 


This  guide  is  the  result  of  work  conducted  under  Naval  Air  Development 
Center  Contract  No.  N62269-76-C-0434  between  1 July  1976  and  30  June 
1977.  Whereas  the  objective  of  this  Navy  Manager's  guide  is  to  clarify 
and  expand  on  the  test  and  evaluation  sections  of  MIL-H-46855,  this  is 
only  one  phase  of  an  effort  to  develop  user's  guides  for  the  major  por- 
tions of  MIL-H-46855. 

The  author  is  indebted  to  the  following  persons  for  their  guidance  and 
contributions: 

CDR  Paul  R.  Chatelier,  Naval  Air  Systems  Command 
Dr.  Lloyd  Hitchcock,  Naval  Air  Development  Center 
Mr.  Ed  L.  Holshouser,  Pacific  Missile  Test  Center 
LCDR  William  F.  Moroney,  Pacific  Missile  Test  Center 

Within  the  Boeing  Aerospace  Company,  the  program  was  directed  by  Mr.  W.  J. 
Hebenstreit  of  Logistics  Support  and  Services'  Crew  Systems  Organization. 
Other  Boeing  personnel  whose  consultation  during  the  program  contributed 
significantly  to  the  information  contained  herein  include:  Mr.  D.  W.  Jahns, 

Dr.  G.  R.  Gardlin,  and  Mr.  M.  M.  Nagata. 
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1.0  INTRODUCTION 


During  the  course  of  system  development,  test  and  evaluation  (T&E)  are 
required  to  assure  that  the  evolving  system  meets  a myriad  of  requirements 
ranging  from  reduction  of  technical  risks  through  demonstrating  the  sys- 
tem's military  utility.  Generally,  two  broad  categories  of  test  and  evalu- 
ation are  used  (Reference  1):  Development  Test  and  Evaluation  (DT&E)  and 

Operational  Test  and  Evaluation  (OT&E),  where  the  former  is  planned,  con- 
ducted, and  monitored  by  the  Developing  Agency  (DA)  of  the  system,  and  the 
latter  is  the  responsibility  of  the  Operational  Test  and  Evaluation  Force 
(0PTEVF0R).  A third  category.  Production  Aceptance  Test  and  Evaluation 
(PAT&E),  is  testing  conducted  on  production  items  to  demonstrate  that  sys- 
tems meet  contract  requirements  and  specifications. 


The  first  two  categories  of  T&E  are  defined  in  Reference  1 as  follows: 

"DT&E  is  that  test  and  evaluation  conducted  to:  demonstrate  that 
the  engineering  design  and  development  process  is  complete;  demon- 
strate that  the  design  risks  have  been  minimized;  demonstrate  that 
the  system  will  meet  specifications;  and  estimate  the  system's  mili- 
tary utility  when  introduced..." 

"OT&E  is  that  test  and  evaluation  conducted  to  estimate  the  prospec- 
tive system's  military  utility,  operational  effectiveness,  and  opera- 
tional suitability  (including  compatibility,  interoperability,  relia- 
bility, maintainability,  and  logistic  and  training  requirements), 
and  need  for  modifications.  In  addition,  OT&E  provides  information 
on  organization,  personnel  requirements,  doctrine,  and  tactics.  Also, 
it  may  provide  data  to  support  or  verify  material  in  operating  instruc- 
tions, publications,  and  handbooks..." 

Human  factors  engineering  (HFE)  is  an  extremely  important  aspect  of  the  total 
T&E  effort.  HFE  T&E  must  be  established  to: 

a)  demonstrate  conformance  of  system,  equipment  and  facility 
design  to  human  engineering  design  criteria; 

b)  confirm  compliance  with  performance  requirements  where  man 
is  a performance  determinant; 

c)  secure  quantitative  measures  of  system  performance  which  are  a 
function  of  man-machine  interaction;  and 


^OPNAVINST  3960.10.  Department  of  the  Navy.  Office  of  the  Chief  of  Naval 
Operations,  "Test  and  Evaluation',  Washington,  D.  C.,  October  1975. 
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d)  determine  whether  undesirable  design  or  procedural  features 
have  been  introduced. 

1.1  Purpose  of  the  Guide 

The  objective  of  this  guide  is  to  provide  assistance  to  human  factors  en- 
gineering (HFE)  Navy  Managers  in  the  planning,  scheduling,  and  perform- 
ance of  human  factors  engineering  T&E  in  accordance  with  the  requirements 
of  MIL-H-46855  (Ke^erence  2).  The  requirements  of  paragraph  3.2.4  through 
3. 2. 4. 3 of  MIL-H-46855  are  interpreted  and  delineated  in  term  of  the  HFE 
T&E  steps  and  techniques  which  need  to  be  performed.  This  guide  will  help 
to  ensure  that  the  HFE  manager  includes  all  appropriate  aspects  of  the 
total  HFE  T&E  process  in  system  development  efforts. 

The  particular  need  for  this  T&E  guide  has  been  to  select  effectively,  for 
a given  type  of  program,  just  which  HFE  T&E  techniques  should  be  performed, 
when  and  how  to  perform  them,  how  their  results  will  be  used,  and  what 
their  relative  cost  will  be.  These  technique  selection  criteria  enable  the 
Navy  manager  to  develop  realistic,  objective  planning  within  the  monetary 
and  time  constraints  of  a program.  This  document  provides  guidelines  for 
the  HFE  manager  that  are  based  on  both  Navy  and  industry  surveys  of  T&E 
experience  and  needs.  The  guide  lists  and  explains  the  basis  criteria  for 
choosing  various  T&E  techniques.  It  provides  the  basis  for  matching  partic- 
ular techniques  to  particular  applications,  and  it  describes  in  general  how 
to  use  the  various  techniques. 

This  guide  may  be  used  at  all  levels  of  program  design  and  procurement, 
including  analysis;  however,  it  is  designed  more  specifically  for  the  Navy 
manager  to  use  during  the  later  design  development  test  and  evaluation 
stages. 


"MIL-H-46855,  Human  Engineering  Requirements  for  Military  Systems,  Equipment 
and  Facilities,  2 May  1972. 
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1.2  Scope  of  the  Guide 

This  document  assumes  Navy  manager  knowledge  of  basic  HFE  concepts.  It 
presents  the  significant  aspects  of  HFE  T&E  alone  rather  than  material 
01  the  complete  field  of  human  factors  engineering.  A complete  descrip- 
tion of  human  factors  engineering  as  well  as  its  typical  applications  and 
value  may  be  found  in  several  HFE  textbooks.  A detailed  description  of 
the  analysis  sections  of  MIL-H-46855  is  contained  in  References  3 and  4. 
These  two  analysis  guides  have  been  prepared  as  part  of  a total  effort 
to  amplify  and  interpret  the  requirements  of  MIL-H-46855. 

This  guide  contains  a description  of  specific  HFE  T&E  tasks  and  techniques. 
It  indicates,  in  general,  how  to  go  about  implementing  or  using  these  tasks 
and  techniques  and  the  best  time  to  use  them.  Comments  are  included  re- 
garding general  technique  selection  and  usefulness. 

The  guide  will  be  of  use  in  preparing  RFP’s  (Request  for  Proposals)  and  in 
the  review  of  HFE  Test  and  Evaluation  Plans  and  HFE  Program  Plans  in  partic- 
ular. A program  planning  decision  tree  checklist  for  Navy  HFE  managers  is 
included  in  Section  2.3  of  this  guide.  This  decision  tree  checklist  indi- 
cates the  decisions  to  be  made  and  actions  to  be  taken  in  preparing  an  RFP 
or  during  the  monitoring  of  the  MIL-H-46855  T&E  effort. 

A similar  guide  to  this  has  been  prepared  for  use  by  HF  engineers  (Reference 
5).  It  includes  most  of  the  same  material;  the  major  difference  in  that 
guide  is  in  additional  detail  presented  in  the  sections  on  HFE  T&E  tasks 
and  techniques.  The  section  on  program  planning  in  the  HF  engineer's  guide 
is  modified  appropriate  to  the  HF  engineer's  point  of  view. 


0 

Geer,  C.  W. , Navy  Manager's  Guide  for  the  Analysis  Sections  of  MIL-H-46855, 

D1 80- 19476-2,  Boeing  Aerospace  Company,  Naval  Air  Development 
Center,  June  1976. 

^Geer,  C.  W.,  Analyst's  Guide  for  the  Analysis  Sections  of  MIL-H-46855, 

D 1 80- 19476-1 , Boeing  Aerospace  Company,  Naval  Air  Development 
Center,  June  1976. 

^Geer,  C.  W. , User's  Guide  for  the  Test  and  Evaluation  Sections  of  MIL-H-46855, 
D194-10006-1 , Boeing  Aerospace  Company,  Naval  Air  Development 
Center,  June  1977. 
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In  Appendix  A there  is  a list  of  all  the  various  organizations  that  were 
contacted  in  order  to  determine  the  kind  of  data  which  should  be  contained 
in  the  two  guides.  These  individuals  were  interviewed  directly  as  to  their 
T&E  experience,  problems,  and  needs.  The  appendices  also  contain  informa- 
tion on  definitions/acronyms  pertaining  to  this  guide,  and  the  T&E  sections 
of  MIL-H-46855. 
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2.0  HFE  TEST  AND  EVALUATION  APPROACH 

This  section  of  the  guide  describes  a general  approach  to  the  HFE  T&E 
effort.  It  includes  both  the  formal  and  the  practical  (or  technical) 
reasons  for  performing  HFE  T&E.  The  approach  to  this  HFE  T&E  effort  is 
detailed  by  a program  planning  decision  tree  diagram  and  by  a section 
describing  the  basic  HFE  T&E  technique  considerations  of  a)  data  inputs, 
b)  timing,  C/  level  of  evaluation  detail,  and  d)  evaluation  applications. 

2.1  Formal  Requirements 

The  formal  requirements  for  performing  test  and  evaluation  are  found  in 
certain  Department  of  Defense  (DoD)  directives  and  SECNAV  (Secretary  of 
Navy),  NAVMAT  (Chief  of  Navy  Material),  and  OPNAV  (Chief  of  Naval  Opera- 
tions) instructions.  The  particular  formal  requirements  for  HFE  and  HFE 
T&E,  are  found  in  NAVMATINST  3900.9  (Reference  6).  Additional  require- 
ments for  HFE  T&E  are  also  found  in  MIL-H-46855.  Two  Data  Item  Descrip- 
tions (DID's)  have  been  developed  for  use  in  performing  HFE  T&E,  There 
are  also  several  guides,  handbooks,  and  general  literature  sources  on  the 
subject  of  HFE  T&E.  Some  of  the  most  important  formal  HFE  T&E  require- 
ments are  always  to  be  found  in  the  program  contract,  including  the  sys- 
tem specification  and  statement-of-work.  While  the  military  is  largely 
controlled  by  directives  and  instructions,  contractors  are  controlled  by 
military  specifications/standards  and  DID's.  Guides,  handbooks,  general 
literature  and  the  attitudes  of  both  the  procurement  agency  and  contractor 
tie  military  and  contractor  HFE  T&E  efforts  together. 

2.1.1  T&E  Directives  and  Instructions 

In  1971,  the  Deputy  Secretary  of  Defense  promulgated  the  policy  for  major 
defense  system  acquisition  via  DoD  Directive  5000.1,  "Major  Systems 
Acquisition"  (Reference  7).  Included  within  this  directive  is  the 


^ NAVMAT INST  3900.9,  Department  of  Navy,  Headquarters  Naval  Material  Command, 
"Human  Factors",  Washington,  D.  C.  20360,  September  1970. 

7 DoD  Directive  5000.1,  Department  of  Defense,  "Major  Systems  Acquisition", 
Washington,  D.  C.  20301,  January  1977. 
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requirement  to  formalize  and  incorporate  T&E  into  the  early  program  devel- 
opment stages.  This  directive  also  required  the  Navy  to  upgrade  and  ex- 
pand T&E  of  all  weapon  systems  to  include  testing  in  a simulated  combat 
environment  as  a prerequisite  for  procurement.  SECNAVINST  5000.1  imple- 
mented this  system  acquisition  policy  within  the  Navy.  Reference  8 pre- 
sents the  Defense  System  Acquisition  Review  Council  (DSARC)  review  require- 
ments for  weapons  systems  acquisition,  and  translates  the  review  checklists 
(DSARC  milestones)  into  HFE  requirements. 

DoD  Directive  5000.3,  "Test  and  Evaluation"  (Reference  9)  establishes  pol- 
icy for  the  conduct  of  test  and  evaluation  by  the  Military  Departments  and 
Defense  Agencies.  It  directs  the  developing  agency  of  the  DoD  component  to 
initiate  T&E  as  early  as  possible  and  to  continue  throughout  the  system  ac- 
quisition process  as  necessary  to  assist  in  progressively  reducing  acquisi- 
tion risks  and  in  assessing  military  worth.  In  addition,  DoD  Directive 
5000.3  requires  that  the  acquisition  schedules  be  based  upon  accomplished 
T&E  milestones  prior  to  the  time  key  decisions  are  to  be  made,  which  would 
commit  significant  additional  resources. 

NAVMATINST  3960.6,  "Planning  and  Implementation  of  Tests  and  Evaluations  of 
New  Weapon  Systems",  (Reference  10;  provides  guidance  for  the  planning  and 
implementation  of  tests  and  evaluations  required  as  a part  of  the  system 
acquisition  process.  This  instruction  provides  guidance  for  preparing  the 
Test  and  Evaluation  Master  Plan  (TEMP). 

0PNAVINST  3960.10,  "Test  and  Evaluation",  (Reference  1)  establishes  policies 
for  T&E  in  Navy  acquisition  programs  and  defines  T&E  responsibilities 


^Holshouser,  E.  L.,  Translation  of  DSARC  Milestones  into  Human  Factors  En- 
gineering Requirements,  Pacific  Missile  Test  Center,  Point  Mugu,  Calif., 
93042,  TP-75-58,  AD-3006927L,  September  1975. 

^DoD  Directive  5000.3,  Department  of  Defense,  "Test  and  Evaluation",  Wash- 
ington, 0.  C.  20301,  January  1973. 

^NAVMATINST  3960.6,  Department  of  Navy,  Headquarters  Naval  Material  Com- 
mand, "Planning  and  Implementation  of  Tests  and  Evaluations  of  New  Weapons 
Systems",  Washington,  D.  C.  20350,  August  1973. 
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for  various  organizations.  It  establishes  procedures  for  planning,  con- 
ducting, and  reporting  T&E.  It  describes  the  relationship  between  develop- 
mental and  operational  T&E  and  it  establishes  procedures  and  a format  for 
test  and  evaluation  master  plans  (TEMP's).  It  also  establishes  procedures 
for  obtaining  fleet  RDT&E  (Research  Development  Test  and  Evaluation)  Research 
and  Development  (R&D)  that  is  not  part  of  an  acquisition  program.  Three 
distinct  phases  of  T&E  are  defined  by  this  instruction.  The  relationship 
between  these  phases  is  discussed  in  Section  3.0. 

NAVMATINST  3900.9,  "Human  Factors",  establishes  policies  and  requirements 
necessary  to  insure  adequate  development  of  human  factors  aspects  of  sys- 
tems and  equipment.  This  includes  all  development,  test  and  evaluation, 
and  production  programs  and  projects.  This  instruction  states  that  the 
human  element  of  the  Navy  systems  shall  undergo  the  same  development,  test 
and  evaluation  steps  as  equipment  elements  of  the  same  system. 

2.1.2  Military  Specifications  and  Standards 

The  primary  source  for  contractor  HFE  T&E  requirements  is  MIL-H-46855.  A 
copy  of  the  sections  pertinent  to  analysis  is  contained  in  Appendix  C.  Fig- 
ure 2.1-1  shows  each  of  the  major  MIL-H-46855  sections  in  functional  rela- 
tion to  each  other.  Section  3.2.4  "Perform  HE  Test  and  Evaluation"  is  the 
function/section  most  pertinent  to  this  guide.  It  is  divided  into  three 
subsections:  planning,  implementation,  and  failure  analysis. 

Unless  otherwise  contractually  noted.  Section  3.2.4  and  its  subsections  are 
the  detailed  requirements  in  MIL-H-46855  necessary  to  provide  an  adequate 
HFE  T&E  program.  Paragraph  3.2.4  indicates  most  of  the  tasks  which  must  be 
performed  by  contractors.  Paragraph  3.2.4. 1 (Planning)  emphasizes  the  need 
for  HFE  tests  integrated  along  with  other  system  tests  in  order  to  save  the 
cost/duplication  of  completely  separate  testing.  Early  testing,  in  accord- 
ance with  this  paragraph,  will  also  save  the  cost  of  unnecessary  late  sys- 
tem rework.  Paragraph  3. 2.4. 2 (Implementation)  contains  a list  of  test  tasks 
to  be  performed  in  order  to  insure  a complete  HFE  T&E  program.  Paragraph 
3. 2.4. 3 (Failure  Analysis)  emphasizes  the  need  to  investigate  human  error. 
HFE  T&E  techniques  listed  in  later  sections  of  this  guide  may  be  used  to 
accomplish  this  requirement  for  human  failure  analysis. 
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3.2.1 


3.2.2 


3.2.3 


Figure  2. 1- 1,  MIL-H-46855  Section  Functional  Relationships 


The  primary  source  of  HFE  design  criteria  to  test  to  is  in  MI L-STD- 1 472 
(Reference  11).  This  standard  presents  HE  design  criteria,  principles, 
and  practices  to  be  applied  in  the  design  of  systems,  equipment  and  facili- 
ties. The  purpose  of  HFE  T&E  in  regard  to  this  standard  is  to  insure:  oper- 
ator/maintair>er  required  performance  is  achieved;  skill,  personnel,  anu 
training  requirements  are  minimized;  required  reliability  of  personnel -equip- 
ment combinations  is  achieved;  and  the  design  within  and  among  systems  is 
standardized.  There  are  several  other  standards  that  contain  design  cri- 
teria which  may  need  to  be  verified  during  HFE  T&E.  However,  they  are  too 
numerous  to  list  here 
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In  addition  to  MIL-H-46855,  the  following  specifications  are  of  some  im- 
portance to  the  HFE  T&E  effort:  MIL-D-8706  (Engineering  Data  and  Test  Con- 
tract Requirements  for  Aircraft  Weapon  Systems);  MIL-D-8708  (Demonstration 
Requirements  for  Airplanes);  MI L-D- 23222  (Demonstration  Requirements  for 
Helicopters);  MIL-M-8650  (General  Specification  for  Aircraft  Mockups);  and 
MIL-M-18828  (Construction  of  Mockups  for  Guided  Missiles). 

2.1.3  Data  Item  Descriptions 

If  called  for  in  the  system  program  contract,  the  Data  Item  Descriptions 
(DID's)  can  be  extremely  important  to  the  HFE  T&E  effort  performed  by  the 
contractor.  The  two  most  important  DID's  pertaining  to  HFE  T&E  are  DI-H- 
2105,  Human  Engineering  Test  Plan  and  DI-H-2111,  Human  Engineering  Test 
Report.  The  HE  Test  Plan  DID  describes  in  detail  how  to  prepare  the  con- 
tractor's test  plan.  It  describes  the  proposed  approach  taken  for  obtain- 
ing T&E  data.  It  establishes  and  explains  all  standards,  tests,  and  asso- 
ciated analyses.  It  also  establishes  other  means  that  will  constitute  ade- 
quate proof  upon  completion  of  the  development  phase  that  acceptable  levels 
of  human  performance,  time,  accuracy,  and  safety  factors  can  be  achieved 
in  operational  use  under  specified  manning  levels.  Reference  12  is  parti- 
cularly helpful  in  preparing  the  HE  Test  Plan. 

The  HE  Test  Report  DID  describes  how  to  prepare  the  documentation  associated 
with  contractor  tests.  The  test  report  is  to  be  used  by  the  procuring  activ- 
ity to  assure  that  the  man-equipment  interface  requirements  for  the  opera- 
tion and  maintenance  of  the  system  conform  to  the  contractual  requirements. 
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MIL-STD-1472B,  Human  Engineering  Design  Criteria  for  Military  Systems, 


Equipment  and  Facilities,  31  December  1974. 
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2.1.4  Guides,  Handbooks,  and  General  Literature 

This  category  of  HFE  T&E  requirements  is  by  far  the  most  voluminous,  and 
it  includes  several  data  sources  of  note.  The  Department  of  the  Navy  RDT&E 
Management  Guide  (Reference  13)  is  a good  source  of  general  Integrated  Logis- 
tics Support  (ILS)  T&E  requirements.  Reference  14  is  an  excellent  summary 
of  policy  and  procedures  for  HFE  T&E  which  can  serve  as  a useful  guide  for 
gaining  an  overview  of  the  scope  of  the  HFE  T&E  effort.  Reference  15,  Human 
Factors  Evaluation  in  System  Development,  is  an  excellent  textbook  source  for 
HFE  T&E  data.  General  theory,  data  collection,  and  analysis  methods  are 
thoroughly  presented. 

References  16,  17,  18,  and  19  are  also  particularly  useful  documents. 


1 2 

Holshouser,  E.  L.,  Guide  to  Human  Factors  Engineering  General  Purpose  Test 
Planning  (GPTP),  PacTfic  Missile  Test  Center,  Point  Mugu,  Calif.,  93042, 

1st  Iteration,  September  1976. 

^Department  of  the  Navy  Management  Guide,  NAVSO  P-2457  (Rev.  1-75), 

1 January  1975. 

14Holshouser , E.  L.,  Human  Factors  Engineering  Policy  and  Procedures  for 
Test  and  Evaluation  of  Navy  Systems,  TP- 75-15,  Pacific  Missile  Test 
Center,  AD-B006035L,  July  1975. 

15Meister,  D. , and  Rabideau,  G.  F. , Human  Factors  Evaluation  in  System 
Development,  John  Wiley  and  Sons,  New  York,  1965. 

16Askren,  W.  B.,  and  Newton,  R.  R.,  Review  and  Analysis  of  Personnel  Sub- 
system Test  and  Evaluation  Literature,  AFHRL-TR-68-7,  AFHRL,  AFSC,  W-PAFB, 

January  1969.  | 

17Keenan,  J.  J.,  et.  al . , Concepts  and  Practices  in  the  Assessment  of  Human 
Performance  in  Air  Force  Systems,  AMRL-TR-65-168,  AD  625  041,  Aerospace 
Medical  Research  Laboratories,  W-PAFB,  September  1965. 

18Myers,  L.  B.,  et.  al..  Guidebook  for  the  Collection  of  Human  Factors  Data, 

Report  PTB  66-3,  AD  631  023,  HRB  - Singer,  Inc.,  State  College,  Pennsyl- 
vania, January  1966.  ] 

19Rabideau,  G.  F.,  "Field  Measurement  of  Human  Performance  in  Man-Machine 
Systems",  Human  Factors,  December  1964. 
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2.2  Technical  Requirements 

These  are  the  important  "real  world"  requirements  such  as  design  criteria, 
safety,  training,  and  life  support  that  the  HFE  observer/evaluator  needs 
to  ensure  are  verified  as  integral  to  the  system  design  being  evaluated. 

The  system  must  meet  these  requirements  in  order  to  insure  the  adequacy 
of  the  man/machine  interface.  Table  2.2-1  lists  each  of  these  require- 
ments in  a matrix  comparison  with  the  techniques  presented  in  Section  5.0. 

The  purpose  of  this  matrix  is  to  show  quickly  which  techniques  may  be  used 
to  test  and  evaluate  the  system  in  order  to  verify  compliance  with  these 
technical  requirements.  Each  of  these  technical  requirements  is  presented 
in  the  following  subsections  along  with  a brief  summary  of  the  reasons  for 
the  importance  of  their  test  and  evaluation. 

2.2.1  Human  Performance 

All  systems  require  a degree  of  human  performance  in  order  to  function  as 
specified.  In  order  to  meet  system  performance  requirements  such  as  speed, 
maneuverability,  range,  or  turnaround  time,  the  operators  and  maintainers 
must  meet  certain  minimum  requirements  for  performing  their  assigned  tasks. 
These  requirements  are  most  always  in  terms  of  time  to  perform  a task  or 
accuracy  (or  reliability)  with  which  a task  must  be  performed.  The  planned 
operator/maintainer  task  times  and  error  rates  must  therefore  be  verified 
by  HFE  observation  and  test. 

2.2.2  Design  Criteria 

Human  engineering  design  criteria  are  the  kinds  of  requirements  contained  in 
MIL-STD-1472  (Reference  Section  2.1.2).  The  incorporation  of  these  criteria 
into  the  hardware  design  insures  a relatively  high  degree  of  operator/maintain- 
er performance.  These  criteria  are  based  on  several  years  of  experience  as 
to  the  design  features  or  details  that  tend  to  minimize  operator  errors  or 
slow  operator  performance.  Many  of  these  design  requirements  are  based  on 
lab  experiments  and  quantified  comparative  data.  For  each  of  the  basic  sys- 
tem functions,  there  should  be  certain  HE  criteria  that  should  be  incorpor- 
ated into  the  design  and  should  therefore  be  evaluated  to  insure  its  exis- 
tence. General  categories  of  HE  design  criteria  include:  displays,  controls, 
labeling,  workspace  layout,  and  maintainability. 
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2.2.3  Safety 

This  is  a special  category  of  design  criteria  that  is  listed  here  and  in 
other  documentation  separately  in  order  to  add  emphasis  to  its  importance. 

As  is  the  case  with  other  design  criteria,  lab  tests  and  experience  have 
indicated  that  certain  design  requirements  (e.g. , rails  on  stairs)  inher- 
ently increase  the  system  safety. 

2.2.4  Training 

Depending  on  the  organizational  setup  or  "charter"  of  a particular  HFE  T&E 
group,  the  evaluation  of  test  participant  training  may  or  may  not  be  neces- 
sary by  the  HFE  group.  Certainly  training  is  an  important  aspect  of  a total 
system  test.  All  other  HFE  aspects  may  be  properly  provided  for  but  im- 
proper training  of  participants  may  be  sufficient  cause  to  ruin  an  otherwise 
successful  test.  Training  requirements  may  be  evaluated  in  terms  of  type 
duration,  and  detail . 

2.2.5  Personnel  Skill/Quantity 

Closely  associated  with  training  is  the  test  participant  skill  level.  This 
is  a combination  of  his  training  on  other  systems  somewhat  related  to  the 
present  system  being  tested  and  his  aptitude  for  the  type  of  work  he  is  per- 
forming. Quantity  simply  refers  to  the  number  of  test  participants  perform- 
ing each  of  the  operator/maintainer  tasks.  The  fact  that  the  proper  number, 
too  many,  or  too  few  operators  are  assigned  to  a test  task  should  be  ob- 
served and  reported. 

2.2.6  Technical  Publications 

This  technical  requirement  is  similar  to  the  training  requirements  in  that 
it  is  often  not  the  job  of  HFE  observers  to  evaluate.  However,  technical 
publications  (tech  orders  or  job  manuals)  are  a very  important  part  of  the 
total  system  under  test.  In  most  instances,  test  participants  operate  or 
maintain  the  system  under  test  in  accordance  with  procedures  that  have  been 
developed  for  the  particular  system  function.  Just  as  it  is  necessary  that 
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the  prototype  hardware  be  tested  and  evaluated,  so  must  the  "prototype" 
or  preliminary  technical  publications  be  tested.  This  evaluation  should 
include  how  well  the  technical  data  meets  the  requirement  to  provide  easily 
understandable  and  accurate  procedural  data.  Technical  narrative,  dia- 
grams, illustrations,  and  photographs  should  all  be  evaluated. 

2.2.7  Life  Support  Criteria 

Life  support  criteria  is  another  special  category  of  design  criteria.  It 
is  listed  here  separately  from  design  criteria  to  add  emphasis  to  these 
requirements.  Life  support  requirements  include:  atmospheric  conditions, 
noise,  vibration,  shock,  toxicology,  radiology,  lighting,  psychophysiology, 
fatigue,  clothing,  and  personal  equipment.  In  many  man/machine  system  de- 
signs, life  support  requirements  are  relatively  insignificant.  However, 
in  high  performance  systems  and  those  requiring  closed  loop  environmental 
control,  life  support  requirements  are  particularly  critical. 

2.3  Decision  Making  Structure 

There  are  a considerable  number  of  decisions  that  must  be  made  and  actions 
taken  by  the  HFE  observer/evaluutor  during  the  HFE  T&E  program  effort  phase. 
The  decision  tree  fold-out  provided  in  this  section  of  the  guide  (Figure 
2.3-1)  is  designed  to  aid  the  Navy  HFE  manager  in  determining  the  proper 
decisions  and  actions.  Its  further  purpose  is  to  provide  the  HFE  manager  with 
planning  assistance  and  checklist  items  to  ensure  inclusion  of  all  necessary 
tasks  and  data  in  the  T&E  program.  This  diagram  is  designed  to  be  an  ad- 
junct to  the  General  Purpose  Test  Planning  (GPTP)  (Reference  12)  diagrams. 

The  Human  Engineering  Test  Plan  is  developed  as  a result  of  specified  pro- 
gram office  requirements  and  Data  Item  Description  (DID's)  which  are  called 
out  in  program  statements-of-work  (SOW's).  The  GPTP  indicates  how  to  pre- 
pare this  Human  Engineering  Test  Plan  which  is  the  documentation  of  how  the 
HFE  effort  will  be  accomplished.  The  GPTP  diagrams  are  designed  to  assist 
the  contractor  HFE  manager  and  observer/evaluator  in  preparing  the  HE  test 
plan.  The  diagram  in  this  section  is  intended  to  assist  in  the  initial 
phase  of  an  HFE  T&E  effort. 
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The  pertinent  decisions  and  possible  actions  have  been  placed  on  the  flow 
chart  diagram  in  blocks  adjacent  to  other  associated  or  related  decision 
and  action  blocks.  Arrows  are  provided  between  the  blocks  to  indicate  the 
direction  of  sequence  of  task  action.  The  placement  of  several  blocks  on 
a single  page  with  connecting  arrows  allows  the  Navy  manager  to  see  quickly 
relationships  between  each  of  the  tasks  that  he  must  accomplish.  A narra- 
tive description  of  these  same  HFE  T&E  tasks  and  task  relationships  would 
undoubtedly  be  hopelessly  complex. 

2.4  Basic  Considerations 

There  are  several  important  considerations  pertaining  to  the  HFE  T&E  effort 
which  are  in  addition  to  the  appropriate  sections  of  MIL-H-46855,  the  pro- 
gram office  manager's  guidelines,  Navy  policy,  and  the  other  constraints 
indicated  in  Sections  2.1,  2.2,  and  2.3.  These  considerations  consist  of 
the  type  of  data  required  to  start  an  HFE  T&E  effort,  the  timing  as  to  when 
to  perform  the  T&E,  the  level  of  T&E  detail  required,  and  the  type  of  speci- 
fic results  normally  expected  from  the  HFE  T&E  effort.  This  section  pertains 
to  these  basic  considerations  in  relation  to  the  overall  HFE  T&E  effort. 

2.4.1  Data  Inputs 

These  consist  of  criteria  or  human  performance  requirements  that  are  to  be 
determined  or  verified  by  T&E.  They  may  be  system  or  mission  requirements 
established  early  in  the  program.  They  may  be  military  specifications  and 
standards  criteria.  Operator/maintainer  requirements  (and  assumptions)  de- 
veloped out  of  MIL-H-46855  analysis  effort  should  be  available  for  eval- 
uation. Ideally,  these  data  are  available  from  a program  data  retrieval  sys- 
tem (e.g.,  FLAG).  They  may  be  available  from  a technology  other  than  HFE. 

In  any  case,  T&E  should  be  conducted  against  set  test  criteria  (success  cri- 
teria) and  if  the  data  does  not  exist,  it  must  be  generated. 

2.4.2  Timing 

Without  the  proper  scheduling  of  the  HFE  T&E  effort,  it  can  turn  out  to  be 
of  little  use  to  the  system  design.  It  is  not  sufficient  just  to  perform 
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HFE  T&E.  It  is  equally  important  to  demonstrate  that  the  results  of  the 
effort  will  be  completed  or  partially  completed  at  a point  in  the  schedule 
when  it  can  properly  impact  the  system  design.  All  too  often,  HFE  T&E  is 
performed  as  an  after-the-fact  documentation  exercise  or  just  a workaround 
procedure  that  appears  in  a technical  publication.  The  later  the  design 
is  tested,  the  less  chance  there  is  to  modify  the  crew  station  or  other 
man/machine  interface.  Late  findings  of  serious  crew  system  problems  can 
be  extremely  expensive,  both  in  accidents  and  system  redesign, 

2.4.3  Level  of  Detail 

The  level  of  T&E  detail  that  must  be  performed  is  significant  to  the  HFE 
manpower  effort.  If  the  wrong  emphasis  is  given  to  the  amount  of  detailed 
data  gathering,  HFE  T&E  may  be  wasted  by  either  obtaining  data  that  has  no 
use  or  by  failing  to  obtain  sufficiently  detailed  data  to  be  useful.  It  is 
the  job  of  the  HFE  manager/observer  to  decide  what  level  of  T&E  will  lead 
to  worthwhile  data  or  useful  design  criteria.  For  example,  new  system  de- 
signs or  programs  often  contain  requirements  that  are  identical  to  previous- 
ly designed  and  tested  systems.  There  is  probably  no  point  in  repeating  a 
detailed  HFE  T&E  of  all  the  system  functions  and  operator  tasks  that  have 
already  been  evaluated.  It  is  simply  not  cost  effective,  especial ly  when 
new  program  schedules  and  manpower  budgets  are  extremely  limited.  The  old 
system  test  and  evaluation  results  should  provide  all  the  data  that  would 
be  needed  for  that  portion  of  the  new  system  design  criteria. 

The  effort  that  should  be  expended  during  the  new  program  is  on  the  new  and 
potentially  critical  functions  and  tasks.  These  are  the  kinds  of  tasks  that 
appear  to  be  marginal  in  terms  of  operator  reliability,  time  performance, 
or  safety  hazards.  The  level  of  test  and  evaluation  detail  attempted  for 
such  tasks  should  be  as  far  down  as  possible  to  the  subtask  element  (indi- 
cator monitoring/button  pushing)  based  on  program  timing/scheduling. 

2.4.4  Appl ications 

The  end  products  of  the  HFE  T&E  effort  are  to  verify  system  design,  discover 
system  inadequacies,  provide  recommendations  for  design  or  other  system 
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changes,  and  provide  information  for  a data  bank  of  human  performance  and 
crew  systems  design  related  data  to  be  used  on  later  programs.  These  out- 
puts may  be  left  in  the  form  of  the  T&E  technique  worksheets.  More  prob- 
ably, the  outputs  should  be  condensed  and  otherwise  modified  to  make  them 
more  easily  understood  by  the  program  design  personnel  and  others  who  have 
use  of  them.  (See  Reference  10  for  GPTP  guide  for  standardized  formats). 
Table  2.4-1  shows  the  applications  for  data  developed  from  using  the  var- 
ious listed  T&E  techniques. 
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Table  2.4-1,  HFE  T&E  Techniques  Data  Applications 
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CONTINUOUS  OBSERVATION 
SAMPLED  OBSERVATION 
OESIGN  CRITERIA  CHECKLIST 
SPEC  COMP  SUMMARY  SHEET 
T.O.  FUNCT  EVALUATION 

37 

37 

37 

38 
38 

B 

1 

1 

B 

B 

B 

B 

B 

B 

HPR  TESTING 

38 

i i 

■ 

g 

g 

mm 

g 

d 

g 

d 

HFTEMAN 

39 

■X 

d 

G 2/G-5  ANTHROPOMETERS 

39 

H 

IS 

j 

■ 

■ 

d 

d 

TERRAIN  VISIBILITY  OEF 

39 

si 

d 

i 

d 

:: 

ENV  & PERF  MEAS.  EQUIP. 

40 

a 

d 

■ 

d 

d 

d 

d 

d 

d 

MANUAL  SYS  MEASUREM’TS: 

g 

■ 

m 

m 

wm 

g 

m 

d 

SYSTEM  RECORDS  REVIEW 

40 

bb 

d 

d 

d 

d 

HUMAN  INITIATED  FAILURES 

40 

d 

d 

B 

H 

d 

d 

d 

test  part,  history  record 

41 

■ 

II 

d 

d 

d 

d 

d 

d 

d 

MANUAL  INDIRECT: 

g 

■ 

■ 

mm 

mm 

■ 

mm 

mm 

mm 

INTERVIEWS 

41 

d 

d 

d 

d 

d 

d 

QUESTIONNAIRES 

41 

d 

d 

d 

g 

d 

d 

d 

d 

FUNCT  DESC  INVIFDI) 

42 

g 

d 

■§K; 

d 

d 

d 

PAARS 

42 

j^g 

■ 

d 

d 

d 

d 

d 

COOPER-HARPER 

43 

d 

d 

d 

d 

d 

PROBLEM  INCIDENT  REP 

43 

■ 

d 

d 

d 

d 

d 

d 

m 

d 

AUTOMATIC  RECORDING: 

H 

m 

■ 

■1 

m 

d 

H 

d 

g 

MOTION  PICTURES 

43 

X 

d 

11 

d 

d 

d 

SOUND  TAPES 

d 

d 

d 

d 

d 

id 

d 

d 

VIDEO  TAPES 

Cl 

d 

d 

d 

d 

d 

d 

d 

d 

■ 

photography 

44 

a 

B : 

■r 

c 

g 

n 

d 

H 

mm 

EVENT  RECORDING 

44 

■x 

H 

d 

d 

d 

d 

d 

OPREDS 

da 

■ 

d 

d 

D 

d 

d 

d 

SECONDARY  TASK  MON. 

O 

1 

■ 

d 

■ 

d 

■ 

d 

d 

d 

AUTOMATIC  PHYSIOLOGICAL: 

mm 

g 

| 

mm 

■1 

g 

mm 

g 

PI 

PHYSIOLOGICAL  INSTRUMENTATION 

45 

EX 

d 

d 

d 

PHYSICAL  MEASUREMENT 

45 

Bfl 

■ 

d 

1 1 

d 

1 

d 

d 

d 

SIMULATION: 

II 

■ 

| 

■ 

mm 

g 

g 

g 

m 

CGE 

46 

EX 

a 

d 

CAR 

46 

d 

d 

^g 

g 

d 

CAPE 

46 

d 

d 

^^d 

d 

FOVEA 

47 

d 

d 

9 

d 

n 

g 

d 

ONLINE  INTERACTIVE 

47 

d 

m 

d 

d 

d 

d 

d 

d 

d 

24 


3.0  TYPES  OF  HFE  TEST  AND  EVALUATION 


As  indicated  in  the  introductory  section  of  this  guide,  there  are  different 
types  of  T&E.  The  definitions  provided  there  are  precise  since  they  are 
based  on  the  Reference  1 instruction.  There  are  additional  detailed  defini- 
tions provided  in  Reference  1 that  are  included  in  this  section  of  the  guide 
There  are  also  included  herein  the  less  formal  definitions  which  are  based 
on  either  system  program  schedules  or  funding  sources. 

3.1  Informal  vs.  Formal  T&E 

In  attempting  to  develop  an  integrated  HFE  T&E  technology,  one  needs  to 
recognize  that  different  types  of  test  and  evaluation  are  involved  in 
systems  development  (both  in  DT&E  and  OT&E):  One,  called  HFE  "t&e"  (low- 

er case)  is  predominant  in  design  concept  feasibility  efforts  and  in  selec- 
tion from  design  configuration  alternatives.  Design  concept  feasibility 
efforts  are  exploratory  in  nature.  They  are  extremely  dependent  upon  paper 
analysis.  The  observation,  experimentation  and  testing  activities  in  this 
category  are  oriented  to  the  goal  of  developing  new  knowledge.  The  selec- 
tion of  design  configuration  alternatives  is  a resolution  process  in  which 
trade  studies  are  performed  and  evaluated.  The  other  type  of  test  and  eval- 
uation, labeled  HFE  "T&E",  is  predominant  in  system  performance  verifica- 
tion. System  performance  verification  is  performed  with  mockups  and  proto- 
type hardware.  The  objective  which  characterizes  this  category  is  one  of 
determining  whether  a man-machine  system,  as  designed,  meets  its  performance 
requirements.  Systems  performance  verification  is  a much  more  formal  test 
and  evaluation  than  the  lower  case  "t&e".  It  is  this  latter  type  of  T&E 
(capital  letters)  which  is  covered  by  this  guide  since  the  applicable  para- 
graph in  MIL-H-46855  carries  the  heading  "Human  Engineering  i_n  Test  and  Eval 
uation".  At  the  same  time,  HFE  "t&e"  may  be  called  for  (both  exploratory 
and  resolution)  when  system  program  Engineering  Change  Proposals  (ECP's)  are 
required.  The  relationship  between  these  differential  aspects  is  illustra- 
ted in  Figure  3. 1-1 . 

3.2  OPNAVINST  3960.10  Definition 

OPNAVINST  3960.10  (Reference  1)  makes  a formal  distinction  between  three 
types  or  phases  of  T&E.  These  are  Development  Test  and  Evaluation  (DT&E), 
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Figure  3. 7- 1:  Informal  vs.  Formal  Aspects  of  HFE  T&E 


Operational  Test  and  Evaluation  (OT&E),  and  Production  Acceptance  Test 
and  Evaluation  (PAT&E).  These  types  of  T&E  are  shown  in  relation  to 
each  other,  to  their  subphase  elements,  and  to  the  program  phases  and 
milestones  in  Figure  3.2-1.  It  is  interesting  to  compare  this  figure 
with  the  Figure  3.1-1  definition  of  T&E  by  phases. 


3.2.1  DT&E 

DT&E  is  required  for  all  acquisition  programs  and  is  conducted  in  four 
major  phases: 

a)  DT-I  is  DT&E  conducted  during  the  conceptual  phase  to  support 
tfieT program  initiation  decision.  It  consists  primarily  of 
analysis  and  studies  to  derive  the  human  factors/system  require- 
ments. 

b)  DT-I I is  DT&E  conducted  during  the  validation  phase  to  support 
the  full  scale  development  decision.  It  demonstrates  the  design 
risks  have  been  identified  and  minimized.  It  consists  of  veri- 
fying the  results  of  the  special  analysis  and  studies  including 
modeling  and  simulation  on  the  critical  areas  identified  earlier. 
It  is  normally  conducted  at  the  subsystem/component  level,  up  to 
and  including  employment  of  engineering  models  for  final  evalua- 
tion. 

c)  DT-I II  is  DT&E  conducted  during  the  full  scale  development  phase 
to  support  the  first  major  production  decision.  It  demonstrates 
that  the  design  meets  its  specifications  in  performance,  relia- 
bility, maintainability,  supportability,  survivability,  system 
safety,  and  electromagnetic  vulnerability.  This  phase  may  be 
further  subdivided  into  additional  phases,  such  as  contractor 
technical  evaluation  (CTE)  and  formal  Navy  technical  evaluation 
(NTES) . The  final  subphase  of  DT-I 1 1 is  TECHEVAL,  the  purpose  of 
which  is  to  certify  that  the  design  meets  specified  requirements 
and  is  ready  for  OPEVAL  (operational  evaluation). 

d)  DT-IV  is  DT&E  conducted  after  the  first  major  production  decision 
to  verify  that  product  improvements,  or  correction  of  design  de- 
ficiencies discovered  during  OPEVAL,  FOT&E  (follow-on  test  and 
evaluation),  or  fleet  employment,  are  effective. 

3.2.2  OT&E 

OT&E  is  required  for  all  acquisition  programs  except  for  those  programs 
designated  by  Chief  of  Navy  Material.  OT&E  is  subdivided  into  two  major 
categories:  initial  OT&E  (IOT&E),  which  is  all  OT&E  accomplished  prior  to 
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the  first  major  production  decision;  and  follow-on  OT&E  ( FOT&E ) , which  is 
all  OT&E  after  the  first  major  production  decision.  OT&E  is  further  divided 
into  five  major  phases  (3  IOT&E,  and  2 FOT&E): 

a)  OT-I  is  any  IOT&E  that  may  be  conducted  during  the  conceptual 
phase  to  support  the  program  initiation  decision.  Most  acquisi- 
tion programs  do  not  require  OT-I.  However,  when  an  OT-I  is  con- 
ducted, existing  systems  or  modifications  thereto  will  normally  be 
used  to  help  estimate  the  military  utility  of  the  proposed  new  sys- 
tem. 

b)  OT-I I is  IOT&E  conducted  during  the  validation  phase  to  support 
the  full-scale  development  decision.  It  provides  an  early  esti- 
mate of  projected  operational  effectiveness  and  operational  suita- 
bility of  the  system;  initiates  tactics  development;  estimates  pro- 
gram progress,  and  identifies  operational  issues  for  OT -III. 

c)  OT-I 1 1 is  IOT&E  conducted  during  the  full-scale  development  phase 
to  support  the  first  major  production  decision.  OPEVAL  (Opera  - 
tional Evaluation)  is  the  final  subphase  of  the  OT -III.  It  con- 
sists of  a demonstration  of  achievement  of  program  objectives  for 
operational  effectiveness  and  operational  suitability,  and  continu- 
ing tactics  development.  OPEVAL  normally  uses  pilot  production 
hardware,  and  begins  about  one  month  after  completion  of  TECHEVAL 
(Technical  Evaluation)  testing. 

d)  QT-IV  is  FOT&E  conducted  after  the  first  major  production  decision, 
but  before  production  systems  are  available  for  testing.  Normally, 
OT-IV  is  conducted  with  the  same  pre-production  prototype  or  pilot 
production  systems  used  in  OPEVAL.  OT-IV  consists  of  testing  of 
fixes  to  be  incorporated  in  production  systems,  completion  of  any 
deferred  or  incomplete  IOT&E,  and  continuing  tactics  development. 

e)  OT-V  if  FOT&E  conducted  on  production  systems  as  soon  as  they  are 
available.  OT-V  provides  for  a demonstration  of  the  achievement  of 
program  objectives  for  production  system  operational  effectiveness 
and  operational  suitability.  In  addition,  OT-V  includes  OT&E  of  the 
system  in  new  environments,  or  in  new  applications,  or  against  new 
threats. 

3.2.3  PAT&E 

As  previously  indicated,  PAT&E  is  testing  conducted  on  production  items  to 
demonstrate  that  systems  meet  contract  requirements  and  specifications.  There 
are  no  subphases  to  PAT&E. 


3.3  General  Definitions 

Further  definition  of  HFE  T&E  may  be  made  from  the  standpoint  of  the 
money  source  to  pay  for  the  T&E  job,  e.g.,  if  design  engineering  money 


pays  for  an  evaluation  of  some  item  of  hardware,  then  the  evaluation  pro- 
cess is  considered  "design".  In  a similar  manner,  the  program  schedule 
helps  define  the  terms,  e.g.,  a paper  analysis  of  a system  concept  may  be 
considered  as  an  on-paper  "evaluation".  However,  since  it  may  occur  early 
in  the  program  during  the  concept  definition  phase,  it  will  most  probably 
be  considered  "analysis"  at  that  time  period. 

The  total  spectrum  of  how  to  categorize  and  differentiate  among  the  diver- 
gent objectives,  criteria  and  applications  of  test  and  evaluation  need  to 
have  a definition  agreed  upon  between  customer  and  contractor  and  between 
customer  agencies  before  significant  HFE  T&E  program  work  may  occur.  A good 
starting  point  for  establishing  a baseline  for  this  would  be  Meister's  Human 
Factors:  Theory  and  Practice  (Reference  20),  and  the  somewhat  earlier  Meister 
& Rabideau  Human  Factors  Evaluation  in  System  Development  (Reference  15). 


20 


Meister,  D.,  Human  Factors:  Theory  and  Practice,  Wiley,  New  York, 


1971. 


30 


4.0  T&E  METHODS 


Numerous  HFE  T&E  techniques  have  been  developed  in  response  to  different 
needs  or  characteristics  that  have  been  required  of  each.  It  is  the  in- 
tent of  this  section  of  the  guide  to  describe  both  the  basic  HFE  T&E  meth- 
ods, or  general  techniques,  and  the  basic  characteristics  that  should  be 
used  to  evaluate  them.  The  methods  are  described  and  evaluated  in  general 
in  relation  to  the  different  qualities  or  characteristics  that  each  typi- 
fies. Also,  the  use  of  mockups,  models,  and  prototype  hardware  are  dis- 
cussed briefly  in  this  section. 

4.1  Technique  Evaluation  Characteristics 

The  choice  of  the  most  efficient  technique  for  performing  an  HFE  T&E  is  a 
critical  step  in  the  total  evaluation  process.  The  general  technique  or 
method  chosen  will  influence  the  entire  conduct  of  HFE  T&E.  Several  gen- 
eral technique  characteristics  should  be  selected  for  use  in  evaluation  of 
the  different  T&E  methods/ techniques.  The  techniques  may  be  compared  to 
each  other  by  use  of  narrative  comments  for  each  evaluation  characteristic 
or  by  use  of  evaluation  worksheets.  The  basic  evaluation  factors  used  to 
choose  the  techniques  should  be  "program  use",  "inherent  features",  "best  use", 
"relative  performance",  and  "program  interface  requirements". 

The  "program  use"  factor  includes  a list  of  the  previous  systems/programs 
that  have  successfully  used  that  method  and  the  applicable  stage  or  phase 
of  a program  that  the  technique  is  best  suited  for. 

The  "inherent  features"  factor  includes  all  of  the  advantages  and  disadvan- 
tages of  that  particular  technique.  The  inherent  validity  or  repeatability 
of  the  technique  should  be  considered  along  with  the  advantages  and  disad- 
vantages. The  implications  of  not  using  the  technique  should  also  be  in- 
cluded. 

The  "best  use"  factor  includes  suitability  for  qualitative  or  quantitative 
data,  testing  as  opposed  to  evaluation,  total  system  versus  subsystem  or 
component  evaluation,  and  single  task  testing  versus  several  simultaneous 
tasks.  The  "relative  performance"  factors  include  time  to  perform,  complex- 
ity, personnel  required,  and  cost. 
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The  "program  interface"  requirements  are  the  input  and  outputs  to/from  the 
techniques  and  the  coordination  required  in  order  to  use  the  technique. 

4.2  General  Technique  Categories 

The  various  HFE  T&E  techniques  are  categorized  for  the  purpose  of  reference 
and  comparison  with  the  previously  indicated  technique  evaluation  character- 
istics. Examination  of  the  various  technique  types  leads  to  the  realization 
that  two  major  categories  are  automatic  and  manual.  The  automatic  techniques 
rely  primarily  on  test  instrumentation  or  computer  simulations.  The  manual 
techniques  rely  to  a large  extent  on  the  observer  as  the  data  collection  de- 
vice. 

4.2.1  Manual  Techniques 

A further  distinction  in  the  manual  category  is  that  of  direct  versus  in- 
direct observation.  System  measurement  is  considered  a third  category  of 
manual  technique;  although  it  could  also  be  classified  as  a form  of  indi- 
rect observation.  Direct  manual  techniques  may  be  further  categorized  as 
being  continuous  or  sampled.  System  measurement  techniques  are  those  in 
which  the  test  evaluator  gathers  HFE  T&E  data  through  a review  of  test  logs, 
maintenance  records,  or  debriefing  records  to  obtain  necessary  HFE  T&E  data. 
Indirect  techniques  include  interviews  and  general  questionnaires.  The  tech- 
nique descriptions  in  Section  5.0  provide  a limited  amount  of  information  in 
the  way  of  individual  technique  comparisons.  The  Reference  5 User's  Guide 
provides  considerable  more  detail  on  each  of  the  techniques. 

4.2.2  Automatic  Techniques 

Automatic  techniques  are  categorized  as  to  those  that  are  primarily  measure- 
ment devices  of  external  data  and  those  that  measure  and  create  new  data 
based  on  specified  inputs.  This  later  category  consists  of  the  computer 
program  techniques  or  computer  simulations.  These  may  be  further  divided 
into  on-line  interactive  techniques  and  off-line  (e.g.,  batch  mode)  tech- 
niques. 

The  measurement  devices  include  all  of  the  recording  techniques/tools  such  as 
video  tapes,  sound  tapes,  event  recorders,  and  photography.  Measurement  de- 
vices also  include  physiological  instrumentation  of  parameters  such  as  heart 
rate  and  EEG.  Also  included  under  measurement  devices  are  the  psysical  an- 
thropological techniques/tools  such  as  the  tape  measure  and  goniometer. 
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4.3  Inherent  Technique/Characteristic  Relationships 

Many  of  the  particular  relationships  between  techniques  and  evaluation 
characteristics  may  be  found  in  Section  5.0.  There  are,  however,  some  gen- 
eral relationships  that  may  be  stated.  One  general  relationship  is  some- 
what similar  to  the  relationship  noted  between  man  (manual)  and  machines 
(automatic)  in  the  notable  Fitts  list  (Reference  4). 

Manual  techniques  tend  to  be  more  subjective  and  qualitative  than  auto- 
matic techniques.  Their  cost  tends  to  be  lower  than  automatic  techniques. 
Automatic  techniques  are  better  used  for  objective  and  quantitative  data. 

The  validity  or  reliability  of  automatic  techniques  tends  to  be  higher  than 
manual  techniques.  This  is  because  manual  techniques  are  often  based  on 
the  opinion  of  the  test  participant  or  observer.  Automatic  techniques  are 
based  on  actual  measured  or  precisely  calculated  data.  Automatic  techniques 
tend  to  be  more  costly  and  more  complex  than  manual  techniques.  Automatic 
techniques  require  more  time  to  initiate  but  once  set  up,  they  are  more 
efficient  than  manual  techniques.  Computer  simulations  (automatic)  are  best 
used  early  in  a program. 

Another  general  relationship  between  techniques  is  the  inverse  relationship 
between  their  fidelity  (or  accuracy)  and  their  ease  of  use.  The  end  product 
of  any  T&E  effort  is  to  make  valid  judgments  in  regard  to  the  real  world 
situation.  Some  techniques  are  inherently  more  precise  than  others  in  terms 
of  their  fidelity  to  the  proposed  actual  operating  system  situation.  Un- 
fortunately, the  reason  for  their  fidelity  is  largely  due  to  the  effort  re- 
quired by  the  use  of  the  technique.  This  inverse  relationship  between  tech- 
nique accuracy  and  ease  of  use  is  illustrated  in  Figure  4.3-1.  The  real 
world  situation  is  represented  at  the  left  of  the  figure.  The  most  accuracy 
would  be  obtained  by  measuring  always  the  real  world  situation.  However,  as 
always,  the  process  of  making  observations  or  measurements  tends  to  distort 
the  test  results  and  a degree  of  test  fidelity  is  lost. 

Field  studies  have  less  fidelity  than  real  world  observations  because  such 
studies  are  set  up  as  artificial  situations.  Simulations  with  the  use  of 
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Figure  4.3-1,  Technique  Fidelity  Vs.  Ease  of  Use 


mockups  cover  a wide  range  on  the  fidelity  scale.  Laboratory  experiments 
also  Lave  a wide  range  of  fidelity  variability.  Some  laboratory  experi- 
ments may  be  quite  realistic  while  others  may  be  highly  abstract.  Com- 
puter or  mathematical  models  are  the  easiest  to  develop,  manipulate,  and 
use,  but  they  are  the  least  accurate  in  their  representation  of  real  world 
situations.  Considerable  advancement  is  now  being  made  in  the  area  of  de- 
veloping computer  models  to  simulate  more  accurately  real  world  situations. 

4.4  Drawings,  Mockups  and  Prototypes 

There  are  a number  of  different  tools  or  media  that  engineers,  in  general, 
and  HF  engineers,  in  particular,  work  with  throughout  the  life  of  a pro- 
gram. These  range  from  engineering  sketches  during  the  exploratory  or  con- 
cept formulation  phase  of  a program  to  the  prototype  hardware  which  would  re- 
sult from  a full  scale  development  effort.  The  techniques  that  the  HFE  ob- 
server/evaluator uses  depend  to  some  extent  upon  the  media  or  engineering 
tools  available. 

The  first  tools  with  which  the  HFE  has  to  work  during  the  "t&e",  or  explor- 
atory, phase  of  the  program  are  engineering  sketches.  These  sketches  may 
be  well  crafted  engineering  drawings  but  are  referred  to  as  sketches  only 
because  of  their  intended  lack  of  contractor  or  customer  sign-off  approval. 
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Engineering  drawings  are  prepared  in  anticipation  of  such  management  approval. 
Interface  control  drawings  are  another  type  of  drawing  that  should  require 
HFE  review.  As  the  name  implies,  these  drawings  are  used  to  describe  and 
to  control  eventually  proposed  interfaces  between  components,  subsystems, 
or  different  contractor's  equipment  items.  Most  of  these  types  of  drawings 
may  be  used  to  perform  a rough  HFE  evaluation,  particularly  by  use  of  the 
design  criteria  checklist.  Software  and  equipment  item  specification  are 
prepared  to  document  required  system  component  performance  and  design  cri- 
teria. To  a limited  extent,  these  may  be  used  to  verify  compliance  with 
contractual  specifications.  HFE  T&E  computer  simulation  techniques  may  be 
used  to  test  system  design  concepts  as  indicated  in  the  drawings  and  equip- 
ment item  specifications. 

As  the  system  design  progresses,  full-scale  mockups  should  be  constructed 
to  evaluate  workspace  and  accessibility  provisions.  Wiring,  cabling,  piping, 
and  ducting  may  be  designed  and  evaluated  much  more  easily  with  mockups. 

It  is  difficult  to  visualize  three-dimensional  problems  from  scaled  down, 
two-dimensional  drawings.  The  mockups  should  be  made  initially  with  the 
easiest  to  use  and  cheapest  material  possible.  Various  thicknesses  of 
plastic  foam  core-filled  cardboard  sheets  may  be  used  quite  easily  with  a 
hot  glue  gun  and  a sharp  matte  knife  to  build  consoles,  racks,  and  even  com- 
plete cockpits.  Console  panel  layout  drawings  may  be  simply  glued  to  the 
foam  core  cardboard  to  simulate  the  appropriately  located  displays  and  con- 
trols. Test  participants  or  evaluators  may  simulate  the  observation  of  dis- 
plays or  actuation  of  controls  by  simply  touching  the  drawing  and  performing 
the  appropriate  hand  (foot)  motion.  As  the  system  design  progresses  and  mock- 
up  tolerances  become  more  critical,  plywood  material  should  be  used.  Plywood 
is  both  more  rigid  and  durable,  although  considerably  more  costly  in  terms 
of  construction  costs.  The  plywood  mockups  may  be  converted  from  a static 
representation  of  the  system  to  a dynamic  or  hot  mockup.  These  mockups  are 
also  referred  to  as  functional  mockups.  The  console  panel  drawings  which 
were  glued  to  the  plywood  may  be  replaced  by  the  actual  displays  and  controls. 
It  is  cheaper  to  develop  a hot  mockup,  which  includes  the  proposed  electrical 
wiring,  than  it  is  to  build  a prototype  with  numberous  design  errors.  A func- 
tional mockup  makes  it  possible  to  study  the  performance  of  personnel  in  sim- 
ulated operational  situations.  The  HF  engineer  can  thereby  evaluate  operating 
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characteristics  of  equipment  in  terms  of  human  performance.  More  realis- 
tic liqhting  and  sound  measurements  may  be  taken.  Procedures  may  be 
verified.  Test  participants  may  be  observed  and  interviewed  with  a much 
greater  degree  of  confidence  as  to  the  validity  of  their  responses.  In 
addition  to  all  of  the  above,  mockups  along  with  photographs  and  movies 
provide  a means  of  design  documentation  to  show  the  evolution  of  the  sys- 
tem configuration. 

The  last  major  engineering  tool  short  of  the  production  hardware  is  the 
prototype.  This  is  the  first  item  of  hardware  produced.  Whereas  it  is 
the  most  valid  representation  of  the  proposed  system  end  item,  it  does 
not  necessarily  comply  with  all  system  requirements.  Depending  on  the  ob- 
jective of  the  system  design,  the  prototype  hardware  may  be  the  first  op- 
portunity to  evaluate  the  design  in  a mobile  state.  With  the  exception 
of  the  computer  simulation  techniques,  virtually  all  of  the  HFE  T&E  tech- 
niques may  be  used  for  prototype  hardware  man/machine  evaluation. 


: 
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5.0  T&E  TECHNIQUES 


This  section  contains  a brief  summary  of  descriptions  of  33  significant 
HFE  T&E  techniques.  They  are  listed  according  to  the  general  categori- 
zation presented  in  Section  4.2.  Additional  details,  including  use, 
validity,  numerous  references  and,  in  some  cases,  examples  of  these  tech- 
niques are  contained  in  Reference  5,  which  is  the  user's  guide  for  the  HFE 
observer/evaluator. 

5.1  Direct  Manual 

5.1.1  Direct  Continuous  Observation 

This  technique  is  simply  the  process  of  taking  a relatively  continuous 
record  of  the  task  or  work  activity  or  some  aspect  of  the  test  per- 
formance. The  operation  may  consist  of  an  observer  keeping  a running 
log  or  description  of  the  test  activity  as  he  understands  it.  The  data 
may  be  recorded  by  hand  on  a clip  board,  or  some  of  the  more  sophisti- 
cated techniques/tools  (Section  5.4.5)  may  be  used  for  recording  events 
and  times. 

5.1.2  Direct  Sampled  Observation 

This  technique  is  identical  to  the  previously  listed  one  with  the  ex- 
ception of  the  amount  of  time  spent  by  the  observer  observing  the  test. 

The  particular  times  chosen  to  perform  test  observation  should,  if  possi- 
ble, be  those  which  coincide  with  the  performance  of  critical  tasks. 

5.1.3  Design  Criteria  Checklist 

The  checklist  is  a series  of  equipment  and  facilities  design  requirements 
or  criteria  taken  from  human  engineering  standards,  e.g.,  MIL-STD-1472, 
handbooks  and  guides.  Often,  during  the  early  stages  of  a program,  a 
checklist  is  develooed  by  HF  engineers  for  that  particular  program.  De- 
sign criteria  which  would  be  applicable  to  the  particular  program  are 
extracted  from  the  various  standards  and  handbooks  and  listed  in  a pro- 
gram unique  checklist.  The  checklists  generally  have  a space  to  the 
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right  of  each  listed  item  of  design  criteria.  This  space  is  divided 
into  three  columns:  compliance,  non-compliance,  and  not  applicable. 

The  HFE  evaluator  reads  the  item  of  criteria,  observes  the  item  of 
hardware  (or  mockup  or  drawing),  and  checks  the  appropriate  space 
for  applicability  and  compliance. 

5.1.4  Specification  Compliance  Summary  Sheet 

This  is  a form  that  is  used  to  verify  that  system  performance  is  in 
accordance  with  specified  HFE  requirements.  Briefly,  the  total  pro- 
cess of  verifyina  HFE  specification  compliance  is:  first,  to  decide 
the  best  method  to  verify  the  specification  requirement  (i.e.,  analysis, 
demonstration,  or  quantitative  data),  second,  to  perform  the  analysis/ 
test  and,  third,  to  document  the  results.  In  any  case,  reports  are 
written  as  to  the  analysis  or  test  results.  The  Specification  Compli- 
ance Summary  Sheet  is  a way  of  summarizing  this  compliance  or  lack  of 
compl iance. 

5.1.5  Technical  Order  Functional  Evaluation 

As  its  title  would  indicate,  this  technique  is  designed  to  evaluate 
technical  orders  or  publications  pertaining  to  the  test.  The  technique 
is  based  on  the  use  of  a form  to  be  completed  by  the  test  observers  while 
they  are  performing  their  other  direct  observations  of  the  test.  The 
technical  publications  must  be  evaluated  as  to  their  usefulness  and  ade- 
quacy in  three  areas: 

a)  Job  Instructions 

b)  Training 

c)  Job  Performance  Aids 

5.1.6  Human  Performance  Reliability  Testing 

The  method  or  aoproach  employed  in  the  use  of  this  technique  is  simply 
direct  observation;  the  purpose  is  to  record  test  participant  errors. 

This  testing  may  occur  at  any  time  during  direct  observation  testing. 
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As  in  the  case  of  the  direct  observation  technique  (Section  5.1.1) 
the  observer  must  first  become  familiar  with  the  anticipated  man/ 
machine  performance. 

5.1.7  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN ) 

HFTEMAN  must  be  considered  as  more  than  an  HFE  TSE  technique.  It  is 
a human  factors  test  and  evaluation  manual  that  is  designed  to  assist 
the  HFE  in  the  areas  of  test  plan  preparation,  test  conduct,  test  data 
evaluation  and  analysis,  and  test  report  preparation.  The  HFTEMAN  con- 
sists of  two  documents:  the  first  contains  detailed  HFE  test  data  and 
the  second  is  a guide  book  supplement  that  contains  specific  HFE  design 
criteria. 

5.1.8  G-2/G-5  Anthropometers 

These  are  two  anthropometric  devices  which  may  be  used  as  models  of 
various  size  or  percentile  crewmen  in  order  to  check  the  adequacy  of 
workstation  geometry.  They  may  be  regarded  as  adjustable  representations 
of  the  more  critical  crew  dimensions.  Each  device  is  more  analogous  to 
a stick  figure  than  to  an  anthropometric  dummy.  They  may  be  adjusted  to 
the  95th,  98th  or  any  desired  percentile  dimensions  for  several  of  the 
more  important  crew  size  parameters. 

The  G-2  anthropometer  may  be  used  at  any  crew  station.  The  G-5  is  de- 
signed primarily  for  use  as  an  escape  envelope  measuring  device. 

5.1.9  Terrain  Visibility  Definition 

With  the  participant  (pilot)  seated  in  the  aircraft  (or  mockup)  cockpit 
and  the  aircraft  in  a horizontal  flight  attitude,  direct  measurements 
of  the  canopy  sill  vertical  and  horizontal  visual  angles  are  made  with 
a surveyor's  transit.  The  total  procedure  establishes  the  following 
four  parameters: 

a)  horizontal  angle 

b)  horizontal  distance  from  pilot's  eye  to  a marked  point 

c)  vertical  distance  from  pilot's  eye  to  a marked  point 
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d)  slant  range  from  pilot's  eye  to  marked  point 
The  outside  visual  field  may  then  be  calculated  from  these  data. 

5.1.10  Environment  and  Performance  Measuring  Equipment 

There  are  several  different  items  of  test  or  measuring  equipment  that 
are  extremely  useful  to  the  HFE  test  observer.  A few  of  these  T&E 
tools  are  included  in  separate  sections,  but  most  are  listed  here: 

a)  Photometer. 

b)  Spot  Brightness  Meter. 

c)  Sound  Level  Meter  and  Analyzer. 

d)  Vibration  Meter  and  Analyzer. 

e)  Thermometer. 

f)  Anemometer. 

g)  Hygrometer  or  Psychrometer . 

h)  Gas  Tester. 

i ^ Forces,  Torque  and  Dimension 
j)  Anthropometry  Instrument  Kit. 

5.2  System  Measurement 

5.2.1  System  Records  Review 

The^e  are  a number  of  typical  test  and  evaluation  program  records  that 
may  be  useful  for  review  by  the  HFE  personnel.  This  technique,  the  re- 
view of  system  T&E  records,  is  unique  in  that  there  is  no  direct  contact 
between  the  test  evaluator  and  the  test  participants.  All  that  is  re- 
quired on  the  part  of  the  HF  engineer  is  to  obtain  permission  to  review  the 
existing  test  records  and  to  go  ahead  with  the  tedious  task  of  look- 
ing through  them.  Typically,  system  records  will  contain  test  loqs, 
maintenance  records,  and  debriefing  records.  The  HF  engineer  may  find 
data  on  equipment  operation  problems,  technical  publication  inadequacies, 
human  initiated  errors.,  and  training  inadequacies. 

5.2.2  Human  Initiated  Failures 

A special  form  or  forms  may  be  provided  to  record  only  test  partici- 
pant errors.  In  order  to  do  this  properly,  the  test  observer  must  be 
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knowledgeable  of  the  system  procedures  to  the  extent  that  he  will  notice 
operator  or  participant  errors.  The  form  requires  the  following  informa- 
tion: where  and  when  the  error  occurred,  a description  of  exactly  what 
the  error  was;  and  estimates  as  to  the  cause  of  the  error,  both  by  the 
observer  and  the  participant.  Possible  causes  may  be  presented  in  a 
short  check  list  format  (e.g.,  fatigue,  environment,  equipment  design, 
etc. ). 

5.2.3  Test  Participant  History  Record 

This  is  not  a direct  test  technique  but  rather  a method  of  improving  the 
test  evaluation  process.  The  Test  Participant  History  Record  form  is 
used  to  collect  data  on  oersonnel  participating  in  HFE  tests.  This  form 
should  be  completed  before  participation  in  the  tests,  is  possible.  Other- 
wise, the  form  may  be  completed  as  part  of  the  post-test  interview.  The 
form  emphasizes  participant  training,  experience  in  systems  similar  to  the 
one  being  tested,  and  participation  in  previous  testing  related  to  the  same 
overall  system  presently  being  tested. 

5.3  Indirect  Manual 

5.3.1  Interviews 

The  interview  technique  is  simply  the  process  of  the  HFE  test  evaluator 
discussing  the  test  events  with  the  test  participants.  This  discussion 
should  be  structured  in  order  to  insure  that  the  most  information  is  ob- 
tained in  the  least  amount  of  time. 

The  product  of  the  interview  is  a quantity  of  test  data  (facts  and  opinions 
to  review  and  evaluate  for  the  purpose  of  presenting  system  problems  and 
recorrmendations,  and  in  many  cases  system  verification. 

5.3.2  Questionnaires 

The  basic  tool  for  obtaining  subjective  data  is  the  questionnaire.  It  is 
the  most  frequently  used  and  most  difficult  to  construct  of  the  subjective 


techniques.  The  questionnaire  provides  a structured  method  for  asking 
a series  of  predetermined  questions  in  order  to  obtain  measurable  ex- 
pressions of  attitudes,  preferences,  and  opinions. 

There  are  four  basic  question  forms  that  may  be  used  in  a questionnaire: 

a)  The  open-end  or  free-answer, 

b)  The  dichotomous  or  two-way, 

c)  The  multiple  choice,  and 

d)  The  rating  scale. 

5.3.3  Functional  Description  Inventory  (FDI) 

The  FDI  is  a new  HFE  T&E  assessment  methodoloqy  or  special  case  question- 
naire. The  FDI  requires  a series  of  investigations  analyzing  the  opera- 
tional functions  of  crewmembers,  with  an  essential  part  involving  the 
determination  of  roles,  duties,  and  tasks  performed  by  each  crewmember. 
Subsequent  to  this,  analysis  time  is  provided  in  order  that  the  crew- 
members can  judge  how  important  these  roles,  duties,  and  tasks  were  for 
mission  success.  They  also  judge  how  frequent  they  are  performed  on 
a typical  mission,  how  adequate  the  training  has  been  to  insure  effect- 
ive performance  of  the  task,  and  finally,  how  effective  the  particular 
system  has  been  in  accomplishing  these  operational  functions. 

5.3.4  Personnel  Activity  Analysis  Radio  System  (PAARS) 

The  Personnel  Activity  Analysis  Radio  System  is  essentially  a direct  ob- 
servation questionnaire  technique.  It  is  based  on  the  use  of  2-way  radios  be- 
tween one  HF  engineer  and  several  maintenance  technicians  or  crews.  It  is 
not  intended  to  be  used  as  an  observation  technique  for  formal  HFE  test  and 
evaluation.  It  is  intended  for  quick  and  efficient  gathering  of  human 
factors  data  from  maintenance  operations  being  performed  on  certain  sys- 
tems in  the  operational  inventory.  Technicians  are  required  to  report 
in  from  the  location  of  their  work.  They  are  asked  questions  in  regard 
to  equipment  problems,  procedural  problems,  delays,  and  potential  hazards. 
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5.3.5  Cooper-Harper  Scale 


The  Cooper-Harper  Scale  is  a well  known  technique  used  to  evaluate 
aircraft  handling  qualities  through  the  use  of  pilot  ratings.  The 
pilot  participants  are  requested  to  provide  their  evaluation  of  given 
flight  tasks,  sub-phases,  phases,  or  even  the  total  flight  (or  mission). 
The  technique  is  basically  dichotomous.  The  pilot  decides  if  the  air- 
craft is  controllable  for  a given  task,  sub-phase,  etc.  If  the  air- 
craft is  controllable,  he  decides  if  the  mission,  phase,  etc.,  per- 
formance is  adequate  with  a tolerable  workload.  If  not  adequate, 
he  choses  (on  a scale  of  three)  how  bad  the  major  deficiencies  are. 

If  adequate,  he  determines  if  the  performance  is  satisfactory  with- 
out the  need  for  improvement.  If  not  satisfactory,  he  determines 
(on  a scale  of  three)  how  bad  the  deficiencies  are.  If  satisfactory, 
he  determines  (on  a scale  of  three)  how  satisfactory  the  performance  is. 

5.3.6  Problem  Incident  Report 

This  form  is  used  whenever  the  data  source  indicates  an  unsatisfactory 
condition  relating  to  the  operator/maintai ner  element  of  the  system  be- 
ing tested.  The  terminology  for  this  technique  or  form  varies.  It  is 
often  called  an  "Unplanned  Event  Record",  "Crew  Interface  Data",  or  just 
a "Squawk  Sheet".  The  report  is  completed  by  HFE  personnel  or  anyone 
observing  HFE  related  problems.  Records  are  kept  on  each  report  as  they 
are  acted  upon  by  HFE  personnel  to  achieve  problem  resolution. 

5.4  Automatic  Recording 
5.4.1  Motion  Pictures 

This  technique  is  simply  the  process  of  filming  participant  performance 
as  a part  of  a system  test.  Actual  prototype  hardware  or  sophisticated 
mockups  should  be  available  to  justify  the  use  of  this  technique.  Less 
sophisticated  mockups  imply  more  uncertainty  in  design,  and  therefore 
a greater  risk  in  expendinq  a motion  picture  effort  on  unsuccessful  con- 
cepts. 
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5.^.2  Sound  Tapes 


Sound  tape  recorders  are  commonly  used  to  maintain  a complete  record  of 
test  conversation  and  events.  Test  notes  may  be  verbally  entered  by  the 
observers  themselves  or  they  may  be  used  to  record  participant  inter- 
view comments.  The  use  of  both  sound  tapes  and  video  tapes  together  is 
frequently  valuable. 

5.4.3  Video  Tapes 

This  technique  is  the  use  of  video  cameras  and  related  equipment  to  make 
video  tape  recordings  for  detailed  review  and  evaluation  of  operator  and 
maintenance  personnel  tasks.  As  with  motion  pictures,  actual  prototype 
hardware  or  extremely  sophisticated  mockups  should  be  available  to  justify 
the  use  of  this  technique. 

5.4.4  Photography 

This  technique  is  perhaps  too  simple  to  be  considered  as  such  and  should 
be  described  rather  as  a HFE  test  and  evaluation  tool.  It  is,  very  simply, 
the  process  of  taking  photographs  of  whatever  tasks,  objects  or  events 
that  are  pertinent  to  the  HFE  effort. 

5.4.5  Event  Recording 

This  is  a technique  for  recording  test  situation  or  event  times.  The 
equipment  involved  in  the  use  of  this  technique  varies  in  complexity  from 
the  stopwatch  to  complete  systems.  Recently  available  recording  equip- 
ment is  designed  such  that  the  test  observer  simply  presses  combinations 
of  keys  to  note  task  functions  as  they  occur.  Data  entries  record  in  a 
solid-state  memory  in  a computer  program  format.  The  data  is  later  trans- 
mitted to  the  computer  by  connecting  the  device  via  a simple  connecting 
cable. 

5.4.6  Operational  Performance  Recording  and  Evaluation  Data  System  (OPREDS) 

OPREDS  is  a real  time  data  acquisition  system  which  presently  is  being  used 
to  evaluate  the  Naval  Tactical  Data  System  (NTDS)  and  related  command  and 
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control  system  performance.  OPREDS  performs  analyses  and  evaluations 
of  ship  systems  in  two  phases: 

a)  Onboard  data  recording  and  preliminary  analysis. 

b)  Data  reduction,  analysis,  storage,  and  retrieval. 

5.4.7  Secondary  Task  Monitoring 

For  the  purpose  of  determining  crew  workload,  test  participants  are 
given  both  operational  tasks  and  secondary  tasks  to  perform.  The  secon- 
dary tasks  are  performed  with  prototype  hardware  or  hot  mockups  on  special 
equipment  that  is  instrumented  through  hardwire  or  telemetry  to  record 
crew  performance.  The  participant  is  instructed  to  perform  the  secon- 
dary tasks  when  not  required  to  perform  the  operational  tasks.  The  crew 
workload  required  to  perform  the  operational  tasks  is  implied  on  the  basis 
of  the  measured  time  (or  effort)  not  spent  doing  the  operational  tasks. 

5.5  Physiological 

5.5.1  Physiological  Instrumentation 

This  instrumentation  may  be  performed  to  monitor  physiological  parameters 
which  will  insure  that  the  test  participant  remains  in  a safe  range  of 
performance.  The  process  of  measuring  test  participant  physiological 
data  is  generally  quite  rigorous.  In  addition  to  all  of  the  set  up  pro- 
cedures required  for  the  test  itself,  it  requires  several  important  tasks 
that  must  be  performed  just  for  the  physiological  instrumentation. 

5.5.2  Physical  Measurement 

This  technique  is  the  process  of  measuring  what  the  test  participants  can 
do  in  terms  of  their  physical  performance  or  what  they  are  doing  in  terms 
of  physical  and  cognitive  performance.  Three  different  types  of  physical 
measurement  may  be  used.  The  first,  anthropometry,  deals  with  potential 
test  participant  physical  performance.  The  other  two,  oculometry  and 
voice  monitoring,  pertain  to  measurement  of  the  participants’  physical 
and  cognitive  processes. 
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5.6 


Simulation 


5.6.1  Cockpit  Geometry  Evaluation  (C6E) 

The  CGE  Program  is  an  experimental  development  partially  funded  by  the 
Joint  Army  - Navy  Aircraft  Instrumentation  Research  (JANAIR)  Program 
Working  Group  to  develop  an  improved  method  of  evaluating  the  physical 
compatibility  of  crewmen  with  specified  crew  stations.  The  heart  of 
the  program  is  a computerized  mathematical  man-model  (BOEMAN).  The  CGE 
Computer  Program  System  (CGECPS)  serves  as  the  basic  integrated  tool  to 
store  required  anthropological  and  geometric  data,  make  computations  to 
simulate  typical  crew  movements  during  the  performance  of  tasks,  check 
and  correct  for  visual  and  physical  interference,  and  output  related  in- 
formation. Crew  station  compliance  with  selected  military  standards  and 
specifications  is  also  performed. 

5.6.2  Crewstation  Assessment  of  Reach  (CAR) 

The  CAR  Model  establishes  the  actual  percentage  of  pilots  that  can  be 
accommodated  in  the  critical  areas  of  a given  crewstation  geometric  con- 
figuration. The  model  examines  hand  and  leg  control  positions,  seat  move- 
ment to  establish  over  the  nose  vision,  and  head  clearance  for  a repre- 
sentative sample  of  the  Navy  pilot  population.  The  model  is  designed  for 
interactive  use;  it  coaches  and  prompts  the  user  in  the  use  of  the  model. 

5.6.3  Computer  Accommodated  Percentage  Evaluation  (CAPE) 

CAPE  is  a Monte  Carlo  computer  model  for  generating  representative  pilot 
anthropometric  features  (including  links)  and  comparing  these  data  with 
an  adjustable  workspace  model  so  that  the  population  accommodated  by  the 
workspace  can  be  estimated  and  maximized.  The  computerized  accommodated 
percentage  evaluation  (CAPE)  model  has  two  options:  exclusion  demonstration 
and  cockpit  analysis. 

An  exclusion  demonstration  determines  what  percentage  of  a potential  popu- 
lation is  excluded  from  a workspace  design  with  respect  to  each  anthropo- 
metric feature  entered  into  the  program.  The  cockpit  analysis  determines 


the  percentage  of  a population  that  will  be  excluded  from  a cockpit 
design  based  on  the  geometric  parameters  of  the  workspace. 

5.6.4  Field  of  View  Evaluation  Apparatus  (FOVEA) 

FOVEA  is  a technique  used  to  test  and  evaluate  the  pilot's  field  of 
view  from  the  cockpit  eye  reference  point.  It  is  designed  to  plot 
the  actual  field  of  view  rather  than  the  field  of  view  as  represented 
by  photographs  and  distorted  reference  grids.  The  technique  requires 
the  procurement  of  certain  test  equipment  such  as  a video  camera,  video 
recorder,  a small  computer,  and  plotter  (or  hiah  quality  graphics  prints). 
Remote  maneuvering  equipment  is  also  required  to  move  the  camera. 

5.6.5  Online  Interactive  Simulation 

The  general  technique  described  in  this  section  pertains  to  the  use  of 
real  time  computer  program  simulations  and  actual  test  participant  opera- 
tors. Like  other  simulations,  online  interactive  programs  are  used  to 
evaluate  and  demonstrate  the  application  of  specific  procedures  and  equip- 
ment to  specific  operations.  The  most  important  requirement  of  an  online 
interactive  simulation  is  that  it  be  an  accurate  representation  of  some 
portion  of  the  proposed  system.  Critical  variables  in  the  proposed  sys- 
tem should  be  properly  duplicated  in  the  simulation. 
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APPENDIX  A:  LIST  OF  PERSONNEL  SURVEYED 


One  of  the  first  tasks  in  the  preparation  of  this  guide  was  to  visit  var 
ious  Navy  labs  and  centers.  Knowledgeable  personnel  were  questioned  in 
regard  to  their  HFE  T&E  experience,  problems,  and  needs.  The  following 
is  a list  of  those  contacted  during  the  survey  visits. 


OPTEVFOR  - Norfolk 
T.  P.  Enderwick 


NADC  - Warminster 
Dr.  L.  Hitchcock 
LCDR  N.  Lane 
D.  E.  Murry 

TECOM/HEL  - Aberdeen 
J.  Perkins 
J.  Miles 
Dr.  J.  Getty 

NATC  - Patuxent  River 
A.  Weaver 

Dr.  S.  G.  Schiflett 

R.  M.  Walchli 

NAVAIR  - Jefferson  Plaza 
J.  C.  Hemingway 

S.  Miller 

CNO  - Pentagon 
Dr.  R.  Smith 

ONR  - Arlington 
Dr.  M.  Tolcott 
G.  S.  Malecki 


NTEC  - Orlando 
LCDR  J.  Funaro 
Lt.  R.  E.  Perryman 
W.  B.  Boney 
R.  G.  Bird 

OPTEVFOR  - North  Island 
Dr.  R.  Rhea 
LCDR  W.  F.  Moroney 
Lt.  W.  R.  Helm 

NELC  - San  Diego 
R.  Coburn 

LCDR  W.  F.  Moroney 
Lt.  W.  Helm 

NPRDC  - San  Diego 
Dr.  D.  Meister 
R.  Harris 

LCDR  W.  F.  Moroney 
Lt.  W.  Helm 
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AFHRL 

APPENDIX  B:  DEFINITIONS/ACRONYMS 

Air  Force  Human  Resources  Laboratory 

AWACS 

Airborne  Warning  and  Control  System 

BOEMAN 

Computerized  Math  Man-Model 

CAPE 

Computer  Accommodated  Percentage  Evaluation 

CAR 

Crewstation  Assessment  of  Reach 

CGE 

Cockpit  Geometry  Evaluation 

CIC 

Combat  Information  Center 

CPS 

Computer  Program  System 

CTE 

Contractor  Technical  Evaluation 

DA 

Developing  Agency 

DACOLS 

Data  Collection  System 

DoD 

Department  of  Defense 

DSARC 

Defense  System  Acquisition  Review  Council 

DT&E 

Development  Test  and  Evaluation 

EEG 

Electroencepholograph 

FLAG 

Feedback  Loop  Action  Generation 

FOT&E 

Follow-on  Operational  Test  and  Evaluation 

FDI 

Functional  Description  Inventory 

FOVEA 

Field  of  View  Evaluation  Apparatus 

GPTP 

General  Purpose  Test  Planning 

HE 

Human  Engineering 

HF 

Human  Factors 

HFE 

Human  Factors  Engineering 

HFTEMAN 

Human  Factors  Test  and  Evaluation  Manual 

HPR 

Human  Performance  Reliability 

ILS 

Integrated  Logistics  Support 

IOT&E 

Initial  Operational  Test  and  Evaluation 

JANA  I R 

Joint  Army-Navy  Aircraft  Instrumentation  Research 

LED 

Light  Emitting  Diodes 
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APPENDIX  B;  DEFINITIONS/ACRONYMS  (cont.) 


NAVMAT 

Chief  of  Navy  Material 

NTDS 

Naval  Tactical  Data  System 

NTE 

Navy  Technical  Evaluation 

OPEVAL 

Operational  Evaluation 

OPNAV 

Chief  of  Naval  Operations 

OPREDS 

Operational  Performance  Recording  and  Evaluation 
Data  System 

OT&E 

Operational  Test  and  Evaluation 

PAARS 

Personnel  Activity  Analysis  Radio  System 

PAT&E 

Production  Acceptance  Test  and  Evaluation 

PSTE 

Personnel  Subsystem  Test  and  Evaluation 

RFP 

Request  for  Proposal 

RID 

Remote  Input  Devices 

SOW 

Statement  of  Work 

SECNAV 

Secretary  of  Navy 

T&E  ' 

Test  and  Evaluation 

TECHEVAL 

Technical  Evaluation 

TEMP 

Test  and  Evaluation  Master  Plan 
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APPENDIX  C:  T&E  SECTIONS  OF  MIL-H-46855 


3.2.4  Human  Engineering  in  Test  and  Evaluation  - The  contractor  shall 
establish  and  conduct  a test  and  evaluation  program  to:  (1)  assure  ful- 
fillment of  applicable  requirements  herein;  (2)  demonstrate  conformance 
of  system,  equipment  and  facility  design  to  human  engineering  design  cri- 
teria; (3)  confirm  compliance  with  performance  reqirements  where  man  is 
a performance  determinant;  (4)  secure  quantitative  measures  of  system 
performance  which  are  a function  of  man-machine  interaction;  and  (5)  de- 
termine whether  undesirable  design  or  procedural  features  have  been  in- 
troduced. (The  fact  that  these  functions  may  occur  at  various  stages  in 
system  or  equipment  development  shall  not  preclude  final  human  engineer- 
ing verification  of  the  complete  system.  Both  operator  and  maintenance 
tasks  shall  be  performed  as  described  in  approved  test  plans  during  the 
final  system  test.) 

3. 2.4.1  Planning  - Human  engineering  testing  shall  be  incorporated  into 
the  test  and  evaluation  program  and  shall  be  integrated  into  engineering 
design  tests,  contractor  demonstrations,  R&E  acceptance  tests  and  other 
major  development  tests.  Compliance  with  human  engineering  requirements 
shall  be  tested  as  early  as  possible.  Human  engineering  findings  from 
early  testing  shall  be  used  in  planning  and  conducting  later  tests. 

3. 2.4. 2 Implementation  - The  human  engineering  lest  and  evaluation  program, 
contained  in  approved  test  plans,  shall  be  implemented  by  the  contractor. 
Test  documentation  (e.g.,  checklists,  data  sheets,  questionnaires,  sche- 
dules, operating  procedures,  test  procedures)  shall  be  available  at  the 
test  site.  Human  engineering  portions  of  all  tests  shall  include,  where 
applicable,  the  following: 

a.  A simulation  (or  actual  conduct  where  possible)  of  mission 
or  work  cycle. 

b.  Tests  in  which  human  participation  is  critical  with  respect 
to  speed,  accuracy,  reliability  or  cost. 
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c.  A representative  sample  of  non-critical  scheduled  and  un- 
scheduled maintenance  tasks. 

d.  Proposed  job  aids. 

e.  Utilization  of  personnel  who  are  representative  of  the  range 
of  the  intended  military  user  population  in  terms  of  skills, 
size  and  strength  and  wearing  suitable  military  garments  and 
equipment  which  are  appropriate  to  the  tasks,  and  approved 
by  the  procuring  activity. 

f.  Collection  of  task  performance  data. 

g.  Identification  of  discrepancies  between  required  and  obtained 
task  performance. 

h.  Criteria  for  the  acceptable  performance  of  the  test. 

3. 2. 4. 3 Failure  Analysis  - All  failures  occuring  during,  or  as  a result  of, 
test  and  evaluation  shall  be  subjected  to  a human  engineering  review  to 
differentiate  between  failures  due  to  equipment  alone,  man-equipment  incom- 
patibilities and  those  due  to  human  error.  The  procuring  activity  shall  be 
notified  of  design  deficiencies  which  contribute  to  human  error. 


